The field of security information and event management (SIEM) is generally concerned with collecting data from networks and networked devices that reflects network activity and/ or operation of the devices, and analyzing the data to enhance security. For example, the data can be analyzed to identify an attack on the network or a networked device and determine which user or machine is responsible. If the attack is ongoing, a countermeasure can be performed to thwart the attack or mitigate the damage caused by the attack. The data that is collected usually originates in a message (such as an event, alert, or alarm) or an entry in a log file, which is generated by a networked device. Exemplary networked devices include firewalls, intrusion detection systems, and servers.
One problem with conventional SIEM services concerns the I/O bandwidth consumed by continually checking individual event records for purging. For example, event records may be checked periodically against each separate retention policy because each policy can delineate varying retention periods for event records. Performance and health event records, for instance, can be retained for just 30 days in contrast to more critical server access and authentication records that can be retained for 1 to 5 years.
At the same time, I/O bandwidth can be strained by queries of event records being serviced by the same log system.
Therefore, what is needed is a robust log system for that selectively purges events with a single pass defined by a union of retention policies.